Papers with monolingual pretraining
Can Cognate Prediction Be Modelled as a Low-Resource Machine Translation Task? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on cognate prediction based on similarities of two languages has not studied their differences or optimized architectural choices. |
| Approach: | They compare statistical and neural MT architectures to a bilingual setup to test their hypothesis . they use monolingual pretraining, backtranslation and multilinguality to test the hypothesis based on the results . |
| Outcome: | The proposed architectures can be used to generate cognates in a given language . the proposed architecture can be employed with monolingual pretraining, backtranslation and multilinguality . |
Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to adapt pretrained Large Language Models to new lower-resource languages are limited to English. |
| Approach: | They propose to combine cross-lingual instruction-tuning with additional monolingual pretraining to adapt LLMs to new lower-resource languages. |
| Outcome: | The proposed model is the first open-source instruction-following LLM for Estonian . the proposed model improves commonsense reasoning and multi-turn conversation capabilities . |
Automatic Evaluation and Analysis of Idioms in Neural Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) struggles with the translation of rare multi-word expressions (MWEs). |
| Approach: | They propose a metric for automatically measuring the frequency of literal translation errors without human involvement. |
| Outcome: | The proposed metric measures the frequency of literal translation errors without human involvement with the models trained in different conditions and across a wide range of metrics and test sets. |
The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining (2026.acl-long)
Copied to clipboard
Jiandong Shao, Raphael Tang, Crystina Zhang, Karin Sevegnani, Pontus Stenetorp, Jianfei Yang, Yao Lu
| Challenge: | Existing research suggests that multilingual large language models can achieve impressive cross-lingual understanding despite largely monolingual pretraining. |
| Approach: | They compare a monolingual-only corpus with a standard web corpus that removes all multilingual documents and then retrain the models from scratch under controlled conditions. |
| Outcome: | The results show that removing bilingual data causes translation performance to drop 56% in BLEU, whereas code-switching contributes minimally. |